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implementation infrastructure to continuously improve teacher practice and impact 
student outcomes. The KDE began its partnership with the SISEP Center in the 

fall of 2014 as a Cohort II Active Implementation State. As one of the first Active 
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This How- 
To-Guide 
provides links 
to modules, 
lessons, activity 
templates, 

and Kentucky 
examples now 
found on the 
AiHub. 


Icons for AiHub 
and Kentucky 
throughout the 
guide indicate 
active links on 
the page. 


Usable Innovation 


Practice Profile 


Fidelity Measure 
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In 2018, the Kentucky Department of Education (KDE) and the State 
Implementation and Scaling-up of Evidence-based Practices (SISEP) 
Center released a white paper, Accomplishing Effective and Durable 
Change to Support Improved Student Outcomes, that describes 

the trials and learning of the KDE, Kentucky's educators and their 
stakeholders as they used the Active Implementation Frameworks 

to improve mathematics outcomes and close persistent educational 
disparities (Ryan Jackson et al., 2018). In this paper the term Kentucky 
is used to represent the co-creative partnership between KDE 
leadership and staff, Kentucky's educators, and their stakeholders 
directly or indirectly involved in the SISEP project. Three follow-up 
publications were cited in the white paper. This paper delivers on the 
commitment to describe the steps Kentucky followed to develop a 
Usable Mathematics Innovation. This How-To-Guide provides links to 
modules, lessons, and activity templates found on the SISEP Active 
Implementation Hub (AiHub). Valuable examples of completed 
activities are provided thanks to the commitment of the KDE, 
Kentucky's educators, and stakeholders for their contribution to the 
SISEP National Community of Practice. 


Overview 


Work began in July of 2014 with a root cause analysis to identify the 
area of focus (mathematics), the population of interest (middle school 
special education students), and the setting (general education core 
instruction). Identifying an area of focus took the dedication of a 
team with diverse perspectives and roles. All levels of the education 
system (state, region, district, school, family and institutes of higher 
education) were represented on the team. The goal of the KDE to 
improve mathematics outcomes was grounded in the State Systemic 
Improvement Plan (SSIP) and the State Identified Measurable Result 
(SiMR): 
“To increase the percentage of students with disabilities 
performing at or above proficient in middle school math, 
specifically at the 8th grade level, with emphasis on 
reducing novice performance...” 


Following the selection of a focus area, a Usable Innovation for 
mathematics was developed consisting of a Practice Profile and a 
Fidelity Measure to clearly define and operationalize the components 
of mathematics instruction that could be easily observed and 
measured in any classroom with a teacher's unique style. The 
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term innovation used in this paper represents Kentucky's Usable 
Mathematics Innovation grounded in the National Council of Teacher 
of Mathematics (NCTM) 8 Mathematics Teaching Practices (NCTM, 
2013). 


Teams were trained on the intentional use of Plan-Do-Study-Act 
Improvement Cycles: Rapid Cycle, Practice Policy Loops, and Usability 
Testing for continuous improvement. Following the completion of 

the Usable Mathematics Innovation, competency teams developed 
training, coaching, and data use systems and measures to 
cvontinuously improve the supports available to teachers and school 
staff. 

Currently, the training, coaching and data-use systems are in Usability 
Testing and will be described in the third white paper. Usability Testing 
is used to proactively test the feasibility and impact of a new way of 
work prior to rolling out the implementation systems and processes 
more broadly. These training, coaching, and data-use systems are 
informed by and aligned to the Mathematics Practice Profile’s core 
components and support districts’ use of their chosen mathematics 
program, curriculum, or set of practices with evidence of effectiveness 
to improve mathematics outcomes (Ryan Jackson et al., 2021). 


“The problem is most districts don’t have 
the time or resources to develop all of the 


systems and measures needed to use an 
innovation effectively — now they have the 
resources and support from their Regional 
Implementation Team.” 
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Active 
Implementation 
research 
(modules, 
lessons, and 
activities) on 
the Active 
Implementation 
Hub are found 
in italics 

and specific 
examples of 
Kentucky's 
completed 
activities are 

in bold italics 
with a Kentucky 
icon. 
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This publication serves as a technical paper or How-To-Guide with 
the intentional step-by-step process Kentucky used to develop a 
Mathematics Usable Innovation. Active /mp/lementation research 
(modules, lessons, and activities) on the Active Implementation Hub 
are found in italics and specific examples of Kentucky's completed 
activities are in bold italics and linked to the AiHub for your use and 
convenience. For a visual timeline including month, year, number of 
participants and meetings, and total hours for completion of each 
component refer to Kentucky's Usable Innovation Timeline. We hope 
these resources will support your organizations development of a 
Usable Innovation. 


“We had so many people involved we have developed a lot 
of capacity, we have become very efficient. It took 6 months 
to develop the Math Practice Profile, 2 months to developa 
coaching Practice Profile with participants using a variety 
of coaching systems, and 2 meetings to develop a Data Use 
Practice Profile — that’s the proof of our capacity.” 

- State Leadership 


Root Cause Analysis 


In August of 2014, the SISEP Center and executive leaders from the 
KDE developed Selection Criteria for the mutual selection of Kentucky 
stakeholders to engage in a comprehensive review of student data in 
the area of literacy and mathematics. The selection criteria allowed 
the KDE to identify a diverse team with the knowledge, skills, and 
characteristics required to complete this task (e.g., facilitation, data 
analysis, content expertise, context knowledge). Once team members 
were selected (n = 22), they created a Terms of Reference that defined 
the team’s purpose, roles, deliverables, and communication loops 

to ensure all members and their stakeholders were informed of the 
team’s progress, from start to finish between team members and 
their stakeholders. Implementation Specialists from SISEP provided 
mini lessons on the use of the Active Implementation Frameworks 

to develop the capacity (e.g., knowledge, skills, abilities) of two 

State Transformation Specialists to lead systems change and use of 
implementation science. 


A Root Cause Analysis was conducted using multiple forms of data. 
For example, state proficiency data for mathematics and literacy, 
drop-out statistics, and College and Career Readiness. The data 
revealed that over 70% of students with disabilities (SWD) were not 
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proficient in elementary school mathematics; more than 80% of SWD 
were not proficient in middle school mathematics; and while 70% of 
SWD graduated from high school in 2014, only 22% were college and 

paid imaravement career ready (Kentucky Performance Rating for Educational Progress 

Syrlee ©) K-PREP, 2013-14). The team used Rapid Improvement Cycles to examine 

the data and come to consensus on middle school mathematics in 
the general education setting as the area of focus and identified 
SWD as the target population. In addition, the team reviewed the 
Teaching, Empowering, Leading and Learning (TELL) survey data 
(Kentucky's working conditions survey). Closing the achievement 
gap, differentiating instruction, integrating technology, and training 
to support SWD were identified as the largest areas of need by 
Kentucky educators (KDE State Systemic Improvement Plan |, 2015). 
The data revealed a need for an implementation and improvement 
infrastructure (training, coaching, data-use support system). Using 
the science of Active Implementation, the team concluded that once 
mathematics instruction was clearly defined and operationalized, 
systems of training, coaching and data use could be developed to 
meet the needs of Kentucky educators. 


Setting a goal for improvement takes time. This first step is critical as 
it identifies an area of focus so that teams at every level of the system 
can develop an infrastructure to effectively implement one thing and 
replicate processes as they are proven to be effective. Development 
of a Usable Innovation and putting it to use by practitioners will move 
an organization from Event Oriented Thinking (root cause event) to 
Systems Thinking (feedback loops to understand and continuously 
improve the system). See Figure 1. 


FIG. 1 Event Oriented Thinking Systems Thinking 


Thinks in straight lines Thinks in ane structure 
Event Oriented 


Thinking A — 
and Systems — c—~>D 
Thinking ) 


In event oriented thinking everything —_In systems thinking a system's behavior 
can be explained by causal chains of emerges from the structure of its 


events. From this perspective the feedback loops. Root causes are not 

root causes are the events starting individual nodes. They are the forces 

the chains of cause and effect, such emerging from particular feedback loops. 

as A and B. Created by Thwink.org 
http://thwink.org/ 


Development of a Usable Innovation and then putting it to use by practitioners will move the organization from 
Event Oriented Thinking (root cause an effect) to Systems Thinking (designing a Usable Mathematics Innovation 
for continuous improvement using Plan Do Study Act Improvement Cycles) (Kainz, 2018). 
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Transformation Zone 


IPAC Team Selection 
Criteria 


IPAC Terms of Reference 


Co-creation of a Mathematics Usable 
Innovation 


In December of 2014, with the support of SISEP the KDE State 
Transformation Specialists, educators, and their stakeholders set out 
to develop a Mathematics Usable Innovation. The KDE recognized 

a quality standard for math instruction was lacking. So, the state 
selected a team of educators to clearly define and operationalize the 
core components of mathematics instruction. Kentucky educators 
developed the first three criteria of a Usable Innovation by agreeing 
on a common philosophy, defining, and then operationalizing 

each core component. The original goal was to identify a few 
mathematics initiatives that were widely used in the state so regional 
partners would have the resources and capacity to provide high- 
quality training and coaching to district and school staff selected to 
participate in Kentucky's Transformation Zone (Ryan Jackson et al., 
2018). In this white paper and in Kentucky, we define initiative as any 
program, curriculum or set of practices a district selects to effectively 
implement Kentucky's Mathematics Innovation grounded in the NCTM 
8 Mathematical Teaching Practices. 


TEAM SELECTION 

The KDE replicated the process used to develop selection criteria 

for the Root Cause team. However, the team selection process was 
more efficient this time because they developed the capacity to form 
a high functioning team. One example was repurposing Root Cause 
Team members who had already received mini lessons on Active 
Implementation. Then, other team members were mutually selected 
for expertise in mathematics instruction, implementation science, 
and knowledge of local district and school context. Consideration was 
also given to selection of diverse perspectives and roles (e.g., school 
and district staff, state mathematics experts, university researchers) 
to generate buy-in, ownership, and collective commitment in the 
Transformation Zone. The selected members were identified as the 
Instructional Practices and Content team (IPAC: n =14). See Example, 
IPAC Team Selection Criteria. Once selection was concluded, the team 
received training on how to complete a Terms of Reference during 
the first meeting in December, 2014. See Example, /PAC Terms of 
Reference. 
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“At the beginning I wondered why I was 
here and then it became clear why we were 
here. The Terms of Reference provided a 
rationale to support communication among 
and across stakeholders.” 


-IPAC Member 


IPAC Infographic & Then, the PAC infographic was developed to conceptualize the work 
of the team and share with the greater community (revised in 2018 
from trial and learning). See Figure 2. The IPAC team was responsible 
for developing the Mathematics Practice Profile that would go on to 
be the foundation for training, coaching and data-use systems and 
measures. 


Instructional Ss. Instructional Practices and Content (IPAC) Team Lay 


Practices and Mission: Provide support to the SSIP team and Regional Educational 
Y Cooperatives by selecting and vetting educational initiatives that are likely to 
Content Team address the gap in mathematics for struggling learners. 


Infographic 


Members with knowledge and expertise in mathematics 


Conductaletate Evaluate each initiative for Select 3-5 
imitativelnvents usability, measurability, and initiatives for 
ry implementation with fidelity menu 


: i A menu of evidence-based practices that will advise districts 
Create a Practice Profile Seater ; 
aA on initiatives that are usable and measurable and are likely to 
address the Gap in mathematics for struggling learners 
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“Our math program selection processes were 
not as thorough as it could have been if we had 
used the Initiative Inventory, Hexagon, and 
Literature review tools. We were thankful that 
our math program aligned with the Kentucky 
Practice Profile and met NCTM criteria.” 


- District Administrator 
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Developing Practice Profiles 


Guidance for Conducting 
Interviews 


KY Math Initiative Inventory 
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Developing a Usable Innovation for Mathematics 
Kentucky relied on the expertise of SISEP liaisons and used resources 


from the AiHub and NIRN website to guide them through the process. 


They read Developing Practice Profiles (Metz et al., 2011) and relied 
heavily on section Ill and V. See, Developing Practice Profiles. 


INITIATIVE INVENTORY 

The purpose of the Initiative Inventory is to guide an organization's 
review of current initiatives that have evidence of impact or in 
Kentucky's case, innovations that are in use and have evidence of 
improving student's mathematics outcomes. The tool is designed 
and used to ensure diverse perspectives through a facilitated 
process. To complete the Initiative Inventory, the workgroup made 
slight modifications to a few items to meet their needs in the area 
of mathematics. Items included: Innovation Name (grade level, 
strategy or focus), Expected Outcome, Target Population, Mandatory/ 
Regulatory, Financial Commitment, State Priority, Level of Success, 
Evidence of Outcomes, Comments. They gathered information on 
each item through informal interviews and document reviews with 
regional, district, and school staff, math practitioners, coaches, 
and university partners, as well as information regarding resource 
allocation and supports for districts and teachers to determine if 
they were sufficient to effectively use the innovation. See example, 
Guidance for Conducting Interviews (Metz, 2016). They identified 30 
widely used math innovations in the state. See Example, Kentucky 
Math Initiative Inventory. 


“It’s automatic now, I was a reluctant participant, but as 
the process evolved it started making more sense. Now, 
we are able to draw upon this work for future reluctant 
educators, to communicate up front how the activities, 
systems, and measures, while all interconnected serve 
multiple purposes.” 

- IPAC Member 
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KY Literature Review 


LITERATURE REVIEW 


The purpose of a literature review is to gather evidence related to 
best practice and to identify the competencies required to use an 
innovation effectively. This workgroup gathered and documented 
literature from peer-reviewed journal articles, mathematics 
organizations, and research underway by Kentucky researchers, 
university and community partners. See Example, Kentucky Literature 
Review. The inclusion of university partners allows teams to have easy 
access to relevant research. Through this process, they also reviewed 
research from the NCTM. This is when the IPAC team realized that 

the data gathered from the Initiative Inventory, informal interviews, 
and literature review converged and began to reveal Kentucky's 
philosophy for teaching mathematics. All of the information collected 
and documented was brought to bear when the team completed the 
next step using the Hexagon Discussion and Analysis Tool. With this 
tool they assessed the 30 widely used mathematics innovations in 
Kentucky schools for fit and feasibility of effective use in districts and 
schools. 


“Using the Hexagon verified for us that there was a 
selection criterion for choosing an innovation.” 


- District Administrator 


HEXAGON DISCUSSION AND ANALYSIS TOOL 

The purpose of the Hexagon Discussion and Analysis tool is to assist 
organizations in the evaluation of new or existing innovations. A 
team with content expertise, diverse perspectives, and knowledge 

of the local context assesses the innovations to determine if they 
meet the need of the identified population, fits and aligns with 

other innovations in use, and that the organization has the capacity 
to sustain use of the innovation (e.g., financial commitment). The 
innovation is also assessed for evidence of effectiveness, supports for 
implementation (e.g., training and coaching supports), and usability 
across a range of contexts (e.g., it is well defined) using a Likert Scale, 
1-5. This information allowed the team to make an informed decision 
regarding the innovations with the strongest evidence for use in 
practice. 
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“Use of the Hexagon allows you to have 
those frank conversations, and it is ok if 
the district math program has a gap as 
long as you are prepared to fill it or select 
something else to push in and use ina 
coordinated effort, it allows districts to 
see where growth is needed.” 


KY Hexagon Discussion 
and Analysis Tool for 
Mathematics 


@) 


-District Administrator 


The entire team gathered to edit the generic discussion and analysis 
questions developed by SISEP liaisons that were associated with 

each component of the Hexagon. They used the information from 
the informal interviews, documents and literature reviews, and the 
NCTM 8 Teaching Practices. The team also established a set of criteria 
and process to review the 30 widely used innovations. First, the 

team only analyzed innovations that served elementary and middle 
school students in general education core instruction, including 
students with disabilities. If an innovation was a commercial textbook 
not aligned with NCTM, an online program, or an intervention for 
targeted or intensive instruction it was removed. Using this criteria, 
23 innovations were removed. Then, three of the remaining seven 
innovations were removed after reviewing the first two categories of 
the Hexagon (Need and Fit) because there was not specific evidence 
that the three innovations addressed the identified area of need 
(students in special education) or that they fit with the state priorities 
(innovation delivered in general education classroom instruction). The 
Hexagon data suggested that the four innovations remaining had a 
high likelihood of consistent use by teachers - there were available 
resources to build capacity for use of the innovations in the state. 

See Example, Kentucky Hexagon Discussion and Analysis Tool for 
Mathematics. 
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“We learned through productive struggle, we had sucha 
struggle that we had to all come together, we all had some 
strong opinions, when we had a strong opinion it made us go 
back and see if it aligned with the research on math and one 
of those NCTM practices. It made us think - everything that 
we train has to be aligned with that Practice Profile, because 
it is Kentucky’s Practice Profile.” 

-IPAC Member 


PRACTICE PROFILE: COMMON PHILOSOPHY AND CORE COMPONENTS 


All of the information gathered (Initiative Inventory, informal 
interviews, document and literature reviews and Hexagon data) 
coalesced to reveal a direct match between Kentucky's philosophy 
for teaching math, innovations in use in Kentucky’s schools, and the 
NCTM 8 Mathematics Teaching Practices. See, Principles to Action 
Executive Summary (2000). The team agreed that the NCTM Practices 
would be the eight core components (or essential functions) of the 
Mathematics Practice Profile. As a result, the first two criteria of a 
Usable Innovation were complete with the development of a clear 
philosophy and identification of core components. 


PRACTICE PROFILE: OPERATIONALLY DEFINING CORE COMPONENTS 
Operationally defining the eight core components for Accomplished 
Use was critical, as well as 3-5 indicators for each core component. 
The operational definitions and indicators serve as a guide to measure 
practice as intended using a walkthrough or fidelity measure. In 
addition, they inform the development of training and coaching 
systems to continuously improve teacher practice. Clearly defining 
the core components and indicators, in operational terms, allows all 
coaches, teachers, and school staff to clearly understand the goal of 
instruction and collectively commit to continuous improvement. 


“Using the Hexagon allows you to make decisions, 
not that you will throw out a math program, 

you can fill the gaps. Maybe ask, do you have the 
coaching supports for a program’s success?” 

- IPAC Member 
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To accomplish this task the IPAC team divided into three groups. 
Each group began to develop a Practice Profile for the innovation 
they had experience developing or using in districts and schools 
for Accomplished Use by teachers (Developmental and Ineffective 
indicators would be the next step). When the three Practice Profiles 
were in the first draft, the teams came together for two 4-hour 
meetings to analyze each Practice Profile’s Accomplished Use 
category. They examined how each team operationally defined the 
eight core components (3-5 item descriptors for each component). 
KY Advice for Reviewing They developed Kentucky Advice for Reviewing Practice Profile 

pentae eines Indicators (NIRN, 2011). They asked: Is the indicator (or activity) a 
priority? If yes, they asked is the indicator measurable, and if not, can 
we make it measurable? Then, they asked does the indicator belong 
somewhere else or should it be removed? It was during their second 
meeting that the three teams realized if Kentucky has a common 
philosophy for teaching math, then the operational definitions should 
remain constant and apply to any mathematics innovation as long as 
it aligned with the eight core components of the Practice Profile. 


“We were assigned to an initiative (math 
program) we had a lot of background in, we a 
wrote three Practice Profiles and came back 

as ateam and realized there was so muchin 
common with all of the Profiles that we only 


needed one Practice Profile aligned to NCT 
- IPAC Member 
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Practice Profile V3 


Vetting and Consensus 
Guidance 


OTISS Items and Hattie 
Effect Size 


For each of the eight components they developed one operational 
definition and associated indicators for Accomplished Use. Finally, 
they operationalized two additional developmental variations 

- Developmental Use and Ineffective Use. The result was one 
common Mathematics Practice Profile for Kentucky - no matter what 
mathematics innovation a district chose. See Example Kentucky 
Mathematics Innovation Practice Profile V3. 


VETTING 

Next, a few team members informally vetted the Practice Profile 
with mathematics experts and staff in a few schools. This was an 
important step to solicit authentic voice from the field and get ready 
to use the Practice Profile in schools and with coaches and teachers. 
The team did not receive any recommendations for revision. This 
may be attributed to the thorough literature and document reviews 
conducted and use of the NCTM Practices as the core components 
of the Practice Profile. See example, Vetting and Consensus Guidance 
(Metz, 2016) for a guide to accomplishing this important step. 


SELECTING A FIDELITY MEASURE 

Selecting a fidelity measure is the last step or component of a Usable 
Innovation. An easy to use fidelity measure aligned with Kentucky's 
Mathematics Practice Profile did not exist, so Kentucky chose to 

use the Observation Tool for Instructional Support Systems (OTISS, 
2015) after on-going deliberation by the KDE executive leaders, using 
feedback from the IPAC and practitioners in the Transformation Zone 
who would be trained to use the fidelity measure. The IPAC team 

was Clear that their goal was to eventually develop a math-specific 
walkthrough that was aligned with their Mathematics Practice Profile. 


“We started with the OTISS fidelity measure 
because there was not a math fidelity measure 
aligned with the Practice Profile.” 


- [PAC Member 


The OTISS is a content and grade level neutral measure that assesses 
general aspects of high-quality instruction to inform supports 

that are or are not in place to support teachers’ effective use of 

an effective mathematics practice (e.g., training, coaching, data 

use support). The research basis for the OTISS is Visible Learning 
(Hattie, 2009), a meta-analysis of over 800 meta-analyses including 
over 50,000 studies related to student achievement. The measure 
consists of seven evidence-based instructional practices that are 
highly correlated with student learning. See Example, O7/SS /tems 


KY Crosswalk, Mathematics 
Practice Profile and OTISS 


“Now we had an example that we could show folks 
- what it means to operationalize a practice, they 
understand it is a tool to help a coach, help ateam 
know what it looks like, sounds like, feels like in 
the classroom.” 


- Regional Implementation Team Member and Coach 
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and Hattie Effect Size. A few members of the IPAC team completed a 
cross-walk of the Mathematics Practice Profile and the OTISS items. 
They found alignment between the OTISS’ seven items and the 

eight components of the Mathematics Practice Profile. See Example, 
Kentucky Crosswalk, Mathematics Practice Profile and OTISS. OTISS 
observations are conducted by trained observers who achieve inter- 
rater reliability. Data is collected and used in aggregate to assess the 
systems of support available to teachers. The OTISS is not used for 
teacher evaluation and individual teacher data is not viewed. The 
aggregate data is lifted up to the Implementation Teams with the 
authority and resources to take responsibility for ensuring effective 
and efficient systems of support are available to teachers. 


DISTRICT SELECTION OF A MATHEMATICS PROGRAM 

The IPAC Team developed short videos to briefly describe the 

three innovations that were widely used in the state however, use 

of the three innovations was not required to participate in the 
Transformation Zone. Some districts already had a mathematics 
innovation they were using that they wanted to strengthen and some 
wanted to explore other mathematics programs or curriculum with 
evidence of effectiveness with the expert guidance of their Regional 
Cooperative. Regional Implementation Teams provide expert support 
as they facilitate a process using the Initiative Inventory and the 
Hexagon Tool to guide a district in the selection of a mathematics 
innovation with evidence and aligned to the Mathematics Practice 
Profile. When a Hexagon factor is scored low, the Regional 
Implementation Team assists the district in strengthening the factor. 
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For example, some districts may not have an on-going training system 
for teachers, so they are supported in securing initial and on-going 
training. Some districts chose to abandon their innovation and chose 
an innovation with readily available training and coaching supports. 
As time goes on, the capacity of regional and district staff to provide 
training and coaching on common mathematics innovations in 
Kentucky has developed and evidence of improved outcomes in 
districts that are using these innovations is documented (Trial and 
Learning in Kentucky's Transformation Zone, forthcoming). 

So, the original goal to develop the capacity of mathematics trainers 
and coaches in Kentucky to effectively use a few proven mathematics 
programs, curriculum, or set of practices may come to fruition as 
capacity is developed and evidence is documented of improved 
student outcomes. 


(66) “The training for teachers and the coach is key so 
they know how to tailor the Practice Profile to the 
grade level.” 

- IPAC Member 


HOW KENTUCKY PUT INTO PRACTICE A USABLE INNOVATION 

The next white paper delivers on the commitment to describe 

How Kentucky Stakeholders Put into Practice a Usable Innovation: 
Measuring the Independent Variable (Forthcoming in 2020-21). See 


Figure 3. 
ar SO: comperency Teams 
Kentucky's 
Usable Math Workgroups with expertise in training (N=9), coaching (N=20), data use (N=18) 
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Doable, and training effectiveness coaching effectiveness 


Easily Assessed in 
Practice ity 


To close the gap Develop a data system to monitor and strengthen training and coaching 
for struggling systems of support that result in high fidelity of any innovation 


students 


OUTCOME 
Training, Coaching, Data system of supports that can be replicated 


for use with any innovation to support teacher practice and improve outcomes 
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The ability to use a common Mathematics Practice Profile and Fidelity 
Measure (Usable Innovation), no matter what program, curriculum 
or set of practices a district may choose, provides a great service to 
districts. When districts choose to be part of the Transformation Zone 
they get expert support from their Regional Implementation Team 
in the selection of a mathematics program aligned with Kentucky's 
Mathematics Practice Profile. Then, they have access to common 
training, coaching, data systems and measures for continuous 
improvement - all developed by Kentucky's stakeholders and 

KY Data Dashboard supported by the KDE in the Kentucky Data Dashboard. Teams can 
easily access and use data at monthly implementation team meetings 
using a PDSA protocol and Rapid Improvement Cycles. See Example, 

KY PDSA Process Kentucky PDSA process. 


“We have had so many people involved we have developed 
alot of capacity, we have become very efficient. It took 6 
months to develop the Math Practice Profile, 2 months to 
develop a coaching Practice Profile with participants using 
a variety of coaching systems, and 2 meetings to developa 
Data Use Practice Profile — that’s the proof of our capacity, 
and sustainability.” 


-State Administrator 


Using lessons learned from use of the OTISS in Transformation Zone 
schools, members of the original IPAC team, mathematics experts 
from Kentucky's Transformation Zone, and community partners 
developed a math specific walkthrough measure in 2019. The 
measure is aligned to the Mathematics Practice Profiles and will be 
Usability Testing during the 2020-21 school year. Kentucky's capacity 
data from Transformation Zone districts and schools demonstrates 
that a district’s schools can enter into Initial Implementation in one 
year, begin to produce improved outcomes for all students, and close 
long-standing disparities for students with disabilities (the target 
population), as well as other disparity groups (KDE State Systemic 
Improvement Plan 111, 2019; Ryan Jackson & Ward, 2019). Now, 
Kentucky's districts can imagine a different future for all students 

as they provide equitable supports to all and close long-standing 
disparities in mathematics outcomes. 
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Lessons Learned 


Throughout the process of creating a Usable Mathematics Innovation, 
the KDE and SISEP liaisons identified key learnings, actions to 
replicate or actions to avoid as teams proactively prepare for the 
development of a Usable Innovation. 


ENGAGE STAKEHOLDERS. 

Engage diverse teams at every level to co-create the Usable 
Innovation so that everyone can see themselves in the work. When 
this was done, there was widespread buy-in and ownership. Ensure 
voice from every level of the organization. In Kentucky, if we had 
collected interview and survey data in a more systematic and 
formalized manner we could have used it to inform future processes 
for engaging stakeholders and prevented the need to return to 
exploration stage activities to gain buy-in. 


COMMUNICATION. 

Bring everyone along in the process. Consistently communicate with 
all stakeholders the rationales for use of all steps in the process. 
Explain why a Usable Mathematics Innovation is critical to supporting 
teachers who improve student outcomes. This supports collective 
commitment for using the innovation. In addition, a communication 
protocol between teams and a communication plan to external 
stakeholders is critical to ensure key stakeholders are informed and 
have a mechanism for providing authentic feedback. This supports 
meaningful, ongoing communication. When communication is 

not strategically planned and documented, it can get lost and key 
opportunities and leverage points for input may be missed. 


TEAM SELECTION. 

Take the time to select diverse participants with a variety of 
knowledge, skills, abilities, and perspectives that are required to 
complete all tasks. Participants will gain tremendous capacity to 
share with new partners in the work. Ensure you are strategic in 
selecting key stakeholders who are representative across the broader 
context where this skill will be used and replicated again and again. In 
Kentucky, there was a teacher and Assistant Superintendent on the 
IPAC team. Having a principal representative would have provided 
the school administrator perspective for the feasibility to support 

the implementation of the essential components of the Usable 
Mathematics Innovation in a school. 


CONTENT USED TO DEVELOP THE USABLE INNOVATION. 
Pay attention to the grain size of the innovation. Choose an innovation 
big enough to build an implementation infrastructure around 
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(training, coaching, data use system), for ease of replication and 
efficiency of time and resources. The team learned that if there is a 
common philosophy for math, then any mathematics curriculum or 
set of practices could align with the Practice Profile. There is no need 
to develop a Practice Profile for each mathematics innovation if a 
common philosophy is established and the core components and 
indicators are clearly defined. 


USABILITY TESTING. 

While the Practice Profile was vetted in a few schools before using it 
in the Transformation Zone districts and schools, the team did not 
formally engage in Usability Testing. The team currently developing 
the math-specific walkthrough made one small change; an additional 
indicator was added in one component. In the coming school year, 
the new math walkthrough tool and the Mathematics Practice Profile 
will go under a rigorous Usability Testing process. The Practice Profile 
is proving to be of sound design as its use is resulting in improved 
student outcomes. As the team reflected on the process, three years 
later, upon developing the math-specific fidelity measure they saw 

a key learning. Identify a proven fidelity measure with evidence of 
effectiveness before trying to develop a new fidelity measure. Without 
the trial and learning of using the OTISS to develop a process for 
collecting, analyzing, and using fidelity data, it would have taken a 
great deal of time and effort to develop the math walkthrough tool. 


TEACHER VERSION OF THE PRACTICE PROFILE. 

In the beginning, teachers were provided a copy of the Practice Profile 
with the Accomplished Use item descriptors only. Then, when coaches 
developed trust with the teachers they began using the Practice 
Profile in its entirety because it allowed the coaches and teachers 

to determine the needed level of support for teacher goal-setting 
(Accomplished Use, Developmental Use, Unexpected Use). Now, 
many coaches are using the practice profile continuum as a way for 
teachers to set their own growth goals. 


FIDELITY MEASURE. 

The OTISS was chosen because there was not a research-based 
fidelity measure for math aligned to the Practice Profile. The 
team acknowledged they did not have the capacity to develop a 
mathematics fidelity measure while they were learning how to use 
the Active Implementation research to develop training, coaching and 
data systems for effective use of a district's chosen math innovation 
in schools. Once the infrastructure was in place and regional teams 
developed the capacity to support districts and their schools, 
Kentucky educators began the process of developing an evidence- 
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“Now, five 
years later, 
coaches who 
are using 
Kentucky’s 
Usable Math 
Innovation 
in schools 
share, “it is 
embedded 
in our work, 
it is part of 
us, it is what 
keeps us 
going.” 

- Regional 
Implementation 
Team Member 


based math fidelity measure (Year 5, or 2018-19). A lesson learned 
was to emphasize the alignment between the OTISS and Mathematics 
Practice Profile early on in the development of the Practice Profile. 
This would have supported more effective use of the OTISS data 
earlier in the process. 


Conclusion 


In summary, an overall key learning is to start small and learn 

how to conduct the Usable Innovation development process with 

a high degree of fidelity. Then, apply the process again and again, 
each time more quickly and efficiently with another content area - 
applying lessons learned for continuous improvement. This learning 
has become embedded into how work is accomplished by the 

KDE and Kentucky stakeholders. In the first Kentucky White Paper, 
Accomplishing Effective and Durable Change to Support Improved 
Student Outcomes, Kentucky's Chief Digital Officer said, “From 

a statewide perspective, everything we do has to be sustainable 

and scalable - has to be. What we needed to do here was rapid 
development, incubate quite a bit, and fail forward with some ideas.” 
Now, five years later, coaches who are using Kentucky's Usable Math 
Innovation in schools share, “it is embedded in our work, it is part of 
us, it is what keeps us going.” 


To improve student outcomes on a useful scale, WHAT is trying to be 
done needs to be teachable, learnable, doable, and easily assessed 
in a typical education setting (Fixsen et al., 2013). Kentucky's Usable 
Math Innovation defines WHAT educators agree they will see in any 
math classroom, no matter what innovation a district may choose 
and no matter what unique style the teacher employs. The goal of 
the SISEP project in Kentucky is to effectively implement the SSIP and 
meet their SIMR or 


“To increase the percentage of students with disabilities 
performing at or above proficient in middle school 
math, specifically at the 8th grade level, with emphasis 
on reducing novice performance (on the State 
Summative Assessment: KPRP).” 


increase in the Proficient scores for SWD over a three-year period 
(SISEP III:4). Summative data from the two elementary schools 
demonstrated a reduction in Novice scores by 10.6 percentage 

points within two years and two middle schools demonstrated a 

14.7 percentage-point decline in the Novice scale for SWD and other 
populations including elementary African American students and 
middle school students receiving Free and Reduced Lunch (SISEP III:4). 


“When we started each school 

was doing what they thought was 
right - using a consistent process 
and curriculum has brought it all 


together.” 


- District Administrator 


NIRN 


FRANK PORTER GRAHAM CHILD DEVELOPMENT INSTITUTE 


NATIONAL IMPLEMENTATION AS | Ss 8 P 
RESEARCH NETWORK 


The KDE anticipated continued growth in proficiency from 

districts in initial implementation during the 2019-2020 school 

year. Unfortunately, summative testing was cancelled due to the 
COVID-19 pandemic as schools prepared for a shift to non-traditional 
instruction. While the shift to non-traditional instruction is a daily 
challenge, Kentucky's educators involved in the SISEP Project continue 
to strengthen and refine use of mathematic practices and their Usable 
Innovation in support of teachers in the classroom (Ryan Jackson et 
al., 2021). 


Usable Innovations provide the content to develop training, coaching 
and data use systems, as well as a mutual selection process to select 
the first set of teachers and staff who will use and support use of 

the innovation. Kentucky replicated the process used to develop the 
Usable Math Innovation to design training, coaching, and data use 
systems and measures. Putting in place a Usable innovation often 
requires a change in the roles, functions, and structures within State 
Education Agencies and the education system in general to more 
efficiently, effectively, and persistently improve effective support for 
teachers and school staff, so they can produce intended outcomes. 
In Kentucky, the responsibility for effective use of a district’s math 
innovation lies on the Implementation Teams who take responsibility 
for effective use of a district's mathematics program (school, district, 
region and state). The onus no longer rests solely on the teacher or 
the school. Rather, any barriers to effective implementation are lifted 
to the team with the authority and resources to remove barriers and 
respond quickly with viable, context relevant solutions. 


i State Implementation and Scaling-up ey 


A How-To-Guide 


23 


Co-creation of Kentucky’s Usable Innovation Process 


References 


Fixsen, D., Blase, K., Metz, A., & Van Dyke, M. (2013). 
Statewide implementation of evidence-based programs. 
Exceptional Children, 79(2), 213-230. 


[ 
Hattie, J. A. C. (2009). Visible learning: A synthesis of over ae \\ \ 
800 meta-analyses relating to achievement. Routledge. 
Kainz, K. (2018). Thinking about data for continuous a ; 
improvement. Presented at the Implementation Science 
Summer Institute, Chapel Hill, NC. 


Kentucky Department of Education (2013-14). Kentucky 
Performance Rating for Educational Progress. https:// 
education.ky.gov/AA/Assessments/Pages/K-PREP.aspx 


Kentucky Department of Education (2015). State 
Systemic Improvement Plan: Phase |. https://education. 
ky.gov/specialed/excep/instresources/Documents/ 
Kentucky%20SSIP%20Phase%20I.pdf 


Kentucky Department of Education (2020). State 
Systemic Improvement Plan: Phase /II:3. https:// 
education.ky.gov/specialed/excep/IDEA/Documents/ 
KentuckySSIPPhaselll3.pdf 


Kentucky Department of Education (2020). State 
Systemic Improvement Plan: Phase /II:4. https:// 
education.ky.gov/specialed/excep/IDEA/Documents/ 
KentuckySSIPPhaselll4.pdf 


A How-To-Guide 


Metz, A., Bartley, L., Blase, K., & Fixsen, D. (2011). Developing Practice 
Profiles. National Implementation Research Network. 


Metz, A. (2015). The Potential of Co-Creation in Implementation Science. 
National Implementation Research Network. 


Metz, A. (2016). Practice Profiles - A Process for Capturing Evidence 
Fale @) ol-1e-18(e)ar-1|74larom lalaren/-1a(elaisrm \t-]a(e)ar-]Miaale)(iaaliale-1d(elama<oi-1-]ge8) 
Nia iela.@ 


National Council of Teachers of Mathematics. (2013). Principles to 
Actions: Executive Summary. Reston, VA. 


Ryan Jackson, K., Fixsen, D., Ward, C., Waldroup, A., & Sullivan, V. (2018). 
Accomplishing Effective and Durable Change to Support Improved 
Student Outcomes. National Implementation Research Network. 


Ryan Jackson, K., Ward, C., Craig, A., Ernst, S., Waldroup, A., & Sullivan, V. 
(2021). How Kentucky Stakeholders Put into Practice a Usable Innovation: 
Measuring the Independent Variable. [Manuscript in preparation]. National 
Implementation Research Network. 


Ryan Jackson, K., & Ward, C. (2019). Tale of Two States: Alignment and 
Cohesion to Close Long Standing Disparities in Student Outcomes. 
National Implementation Research Network. 


N | R N NATIONAL IMPLEMENTATION y Ss | E a) 
RESEARCH NETWORK 
Ah ling-u 
of E 


FRANK PORTER GRAHAM CHILD DEVELOPMENT INSTITUTE 


